Python: Real World Machine Learning by Prateek Joshi & John Hearty & Bastiaan Sjardin & Luca Massaron & Alberto Boschetti

Author:Prateek Joshi & John Hearty & Bastiaan Sjardin & Luca Massaron & Alberto Boschetti
Language: eng
Format: epub
Publisher: Packt Publishing
Published: 2016-11-14T00:00:00+00:00

In this case, we're able to plot the results of a jitter test to easily identify whether a model has overfit. From a very strong initial position, an overfit model will typically rapidly decline in performance as small amounts of jitter are added. For better-fitting models, the loss in performance with added jitter is much reduced, with the degree of overfitting in a model being particularly obvious at low levels of added jitter (where a well-fit model will tend to outperform an overfit counterpart).

Let's look at how we implement a jitter test for overfitting. We use a familiar score, accuracy_score, defined as the proportion of class labels predicted correctly, as the basis for test scoring. Jitter is defined by simply adding random noise to the data (using np.random.normal) with the amount of noise defined by the configurable scale parameter:

from sklearn.metrics import accuracy_score def jitter(X, scale): if scale > 0: return X + np.random.normal(0, scale, X.shape) return X def jitter_test(classifier, X, y, metric_FUNC = accuracy_score, sigmas = np.linspace(0, 0.5, 30), averaging_N = 5): out = [] for s in sigmas: averageAccuracy = 0.0 for x in range(averaging_N): averageAccuracy += metric_FUNC( y, classifier.predict(jitter(X, s))) out.append( averageAccuracy/averaging_N) return (out, sigmas, np.trapz(out, sigmas)) allJT = {}

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.